Repository navigation
driver/redis: an older CLUSTER SLOTS read ending after a newer one no longer overwrites the slot table (v1.6.0, Refs #1219) - #1221
Merged
Conversation
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
…for an older read when the newer one fails; drop the unordered branch (Refs #1219)
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
driver/redis: an older CLUSTER SLOTS read that ends after a newer one no longer overwrites the slot table. Refs #1219 (found by the review of #1215; pre-existing).
The problem
applyTopologyrebuilt the slot table from whateverCLUSTER SLOTSreply reached it, with no ordering. Two overlapping reads could apply out of order: a read that took its table before a slot moved, but ended after a newer read had applied the move, put the slot back to its pre-move owner (extra-MOVEDrounds or a stale route until the next refresh). #1215's shared reads make overlapping reads more likely.The change
topoRefresh.seq, assigned underrefMuwhen the read is registered: driver/redis: sharded PubSub reroutes share one CLUSTER SLOTS read (v1.6.0, Refs #1203) #1211/driver/redis: commands that meet the same MOVED or lost primary share one topology read (v1.6.0, Refs #1214) #1215).runRefreshnow passes it throughreadTopologytoapplyTopology, which, underc.mu, drops a result whose sequence is lower than the last applied one (ClusterClient.topoApplied) and otherwise applies it and records its sequence. A dropped read returns nil (a newer table is in place, so the caller's goal is met; PubSub/command joiners only share reads that started after their trigger, and the newer read is one).topoApplied.SentinelClient.handleSwitchMasterswaps the master client, and it is called only fromwatchFailover, whichsubscribeLoopruns inline: +switch-master notifications are handled one at a time in delivery order, so a slow dial of an older failover cannot install its client after a newer one. There is no second writer. A comment onhandleSwitchMastersays so. (queryMasterat construction is a single read.)Commits:
e2a6695the tests (RED on main),b6958a9the fix.RED / GREEN
RED,
e2a6695alone (macOS arm64,go test ./driver/redis -run 1219 -v):The test uses the fake cluster of #1211: hold read 1 after its table (pre-move) is taken, move the slot, run read 2 to the end (applied, the slot routes to the new owner), release read 1, and assert the table still names the new owner. The second test checks the guard is not too strong: reads that end in the order they started all apply (four moves back and forth), and a read cut short by its caller's context applies nothing and does not block the next one. No sleep and no clock decides a result.
GREEN,
b6958a9: both pass.Mutants (each alone,
go test -run '1219|1214' -count=2)runRefreshpasses 0 (unordered; that branch was removed in the r8 follow-up)Checks
-race -run '1219|1214' -count=20Linux dockergolang:1.27.3--cpus 2: ok; wholedriver/redis-race -count=1: ok (175 s). macOS-race -count=20: ok.GOOS=windows go vet ./driver/redis: clean.golangci-lintv2.15.0GOOS=linux: only the two pre-existingerrcheckfindings in untouched files.Release note
driver/redis (cluster): the slot table is no longer replaced by an older
CLUSTER SLOTSread: a read that finishes after a newer one (registered later) is dropped instead of putting a slot back to its previous owner, which cost extra-MOVEDrounds or a stale route until the next refresh. The order is that of registration, not of the cluster's truth: a read that hits a lagging node can still install an older table, which the next-MOVED(or the 60 s refresh) corrects. The sentinel client is not affected (failover notifications are handled in order).Review r8 follow-up (commit
45a63cb)topologyProbes) after the newer one was sent, can install the older table and the fresher answer is dropped; the reviewer's probe P2 shows head leaving the slot on the old owner where main had it right by luck, corrected by one-MOVED. This is the mirror image of driver/redis: an older CLUSTER SLOTS read that finishes after a newer one overwrites the slot table #1219 and cannot be avoided without a server epoch.TestAnOlderReadStillAppliesWhenTheNewerOneFailed1219(the reviewer's probe P1): an older read carrying the move is in flight, the newer read is cut short, the older read must still apply. Mutants "a failed read advances the order" (inreadTopologyand inrunRefresh) and "drop a result when a newer read is merely registered" fail it (they passed every other test).CLUSTER SLOTShas none;CLUSTER NODESreportsconfig-epochper node. Not done here.45a63cb:-race -run '1219|1214' -count=20Linux docker--cpus 2ok, wholedriver/redis-race -count=1Linux ok (176 s); macOS-raceok (1214 in two batches of 10: 20 runs of the herd tests exhaust the macOS ephemeral ports, an environment limit);GOOS=windows go vetclean; golangci-lint only the two pre-existingerrcheckfindings.